Virtual shadows for enhanced depth perception

ABSTRACT

In order to improve depth perception for an image displayed during a laparoscopic surgery, a representation of a shadow of a tool included in the image and used in the laparoscopic surgery is identified and introduced into the image. A processor augments a three-dimensional (3D) model including a 3D representation of a surface of an object included in the image, and a representation of the tool by introducing a virtual light source into the 3D model to generate a virtual shadow within the 3D model. The processor subtracts the representation of the shadow out of the augmented 3D model and superimposes the representation of the shadow on the image to be displayed during the laparoscopic surgery.

FIELD

The present embodiments relate to a method for generating virtualshadows on an image.

BACKGROUND

Laparoscopic and endoscopic cameras are used to see inside the body of apatient for surgery and diagnostics. Laparoscopic and endoscopic camerasgenerate essentially two-dimensional (2D) images of three-dimensional(3D) anatomy. During a surgery, an operator (e.g., a surgeon) guidestools to target regions inside the body of the patient to, for example,cut tissue, make dissections, and/or take biopsies based on a 2D imagestream generated with the laparoscopic camera and/or the endoscopiccamera. With the 2D image stream, the sense of depth and 3D is lost.This makes navigating and controlling the tools difficult for theoperator.

Current approaches to improving depth perception include creating 3Dvisualizations and introducing missing depth cues such as a motionparallax or a shadow. 3D visualizations are created with a stereolaparoscopic camera, and 3D displays are used. The 3D displays may makethe operator nauseous, and have limited working volumes, making the 3Ddisplays challenging to integrate into an operating room or work withmultiple team members. Also, some 3D displays require the operator towear glasses, which may cause discomfort during long procedures.

Motion parallax and motion perspective may be reintroduced with avirtual mirror. The use of the virtual mirror, however, requiresregistration of a pre-operative model to the patient, and the virtualmirror is configured to be used with augmented reality. Shadows may bereintroduced with additional lights on the laparoscopic camera. Becausethe additional lights are co-aligned with the laparoscopic camera, thestrength of the shadows that are cast is limited and the direction ofthe shadows may not be optimal.

SUMMARY

In order to improve depth perception for an image displayed during alaparoscopic surgery, a representation of a shadow of a tool included inthe image and used in the laparoscopic surgery is artificially createdand introduced into the image. A processor generates a three-dimensional(3D) model including a 3D representation of a surface of an objectincluded within the image with a structured light endoscope. Theprocessor determines a position of the tool and introduces arepresentation of the tool into the 3D model based on the determinedposition. The processor augments the 3D model by introducing a virtuallight source into the 3D environment to generate a shadow projected ontothe 3D model. The processor identifies the representation of the shadowfrom the augmented 3D model and adds the representation of the shadow tothe image to be displayed during the laparoscopic surgery.

In a first aspect, a method for augmenting an image of a surface of anobject is provided. The method includes identifying, by a processor, theimage of the surface of the object. The image of the surface of theobject is an image of the surface of the object and at least a portionof a tool. The processor identifies a three-dimensional (3D) model ofthe surface of the object. The processor generates a first augmented 3Dmodel. The generating of the first augmented 3D model includessuperimposing a representation of the tool on the identified 3D model ofthe surface of the object. The processor introduces a virtual lightsource into the generated first augmented 3D model. The processorgenerates a second augmented 3D model. The generating of the secondaugmented 3D model includes generating a representation of a shadow onthe surface of the object in the generated first augmented 3D modelbased on the virtual light source introduced into the generated firstaugmented 3D model. The processor augments the identified image of thesurface of the object. The augmenting of the identified image of thesurface of the object includes superimposing the generatedrepresentation of the shadow on the identified image of the surface ofthe object and the tool.

In a second aspect, a non-transitory computer-readable storage mediumstores instructions executable by one or more processors to augment animage of a surface of an object. The instructions include identifying,by a processor, the image of the surface of the object. The image of thesurface of the object is an image of the surface of the object and atleast a portion of a tool. The tool is adjacent to the surface of theobject. The instructions further include identifying, by the processor,a 3D model of the surface of the object, and superimposing, by theprocessor, a representation of the tool on the identified 3D model ofthe surface of the object. The instructions include introducing, by theprocessor, a virtual light source into the superimposed representationof the tool and identified 3D model of the surface of the object. Theinstructions further include generating, by the processor, arepresentation of a shadow on the surface of the object within thesuperimposed representation of the tool and identified 3D model of thesurface of the object based on the introduced light source. Theinstructions include superimposing, by the processor, the generatedrepresentation of the shadow on the identified image of the surface ofthe object and the tool.

In a third aspect, a system for generating an augmented image of asurface of an object and at least a portion of a tool is provided. Thesystem includes a memory configured to store an image of the surface ofthe object and at least the portion of the tool, and a 3D model of thesurface of the object. The system also includes a processor incommunication with the memory. The processor is configured tosuperimpose a representation of the tool on the identified 3D model ofthe surface of the object. The processor is also configured to introducea virtual light source into the superimposed representation of the tooland the identified 3D model of the surface of the object. The processoris configured to generate a representation of a shadow on the surface ofthe object within the superimposed representation of the tool andidentified 3D model of the surface of the object based on the introducedlight source, and is configured to superimpose the generatedrepresentation of the shadow on the stored image of the surface of theobject and at least the portion of the tool.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart of one embodiment of a method for augmenting animage of a surface of an object with a virtual image;

FIG. 2 is an example of an image of a surface of an object and a toolbeing used during a medical procedure;

FIG. 3 shows an example of a first augmented three-dimensional (3D)model;

FIG. 4 shows another example of a first augmented 3D model;

FIG. 5 shows an example of a virtual light source introduced into theexample of the first augmented 3D model shown in FIG. 4;

FIG. 6 shows an example of a representation of a virtual shadow afterthe first augmented 3D model has been subtracted from the secondaugmented 3D model;

FIG. 7 shows an example of a generated representation of a virtualshadow fused with an image of a surface of an object and a tool;

FIG. 8 shows one embodiment of an image-guided system; and

FIG. 9 shows one embodiment of an imaging system.

DETAILED DESCRIPTION OF THE DRAWINGS

The present embodiments may be used to introduce a depth cue into alaparoscopic video and thus create an enhanced visualization during, forexample, a laparoscopic surgery. The depth cue may be, for example, avirtual shadow. To generate the virtual shadow, a processor identifies athree-dimensional (3D) model of a tissue surface within a body to beimaged and a position of a tool to be used during the laparoscopicsurgery in a same coordinate system.

The 3D model of the tissue surface is generated with, for example, astructured light endoscope. The structured light endoscope generates apoint cloud of the tissue surface the structured light endoscope isobserving. A processor generates a mesh for the 3D model of the tissuesurface. Alternatively, a stereo endoscope or a monocular scope usingtemporal information may generate the 3D model of the tissue surface.

The position of the tool may be estimated using a tracking system. Thetracking system may include, for example, an optical tracker includingone or more cameras and tracker markers placed on a proximal end of thetool. From positions of the markers and a 3D model of the tool, theposition of the tool in 3D space may be estimated. Through a calibrationprocess, a position of a tip of the tool in the 3D space may beestimated. In order to generate the virtual shadow, the position of thetool relative to the 3D model of the tissue surface is to be known.Accordingly, the structured light endoscope may also include one or moretracker markers.

A number of models of different tools may be generated off-line andsaved in a memory. The processor may identify one of the stored modelsthat corresponds to the tool being used. Alternatively, the processormay determine a shape and a size of the tool based on one or more imagesgenerated by a laparoscope (e.g., the structured light endoscope), andthe processor may generate a model of the tool during the laparoscopicsurgery, for example.

The model of the tool is introduced into the 3D model of the surface ofthe tissue based on the determined position of the tool in the 3D space.A virtual light source is introduced into the 3D model of the surface ofthe tissue. The virtual light source may be positioned in any number ofpositions within the 3D model of the surface of the tissue including,for example, in an up vector from a corresponding laparoscopic cameraimage or at a position upward of gravity as defined by the opticaltracker.

The light from the virtual light source causes a virtual shadow to becast onto the representation of the tissue within the 3D model of thesurface of the tissue. The 3D model of the surface of the tissue may berendered from a viewpoint of the laparoscope to be used during thelaparoscopic surgery (e.g., the structured light laparoscope). The 3Dmodel of the surface of the tissue is rendered with and without theshadow, and the 3D model without the shadow is subtracted from the 3Dmodel with the shadow. This subtraction of the models generates thevirtual shadow. The virtual shadow is then blended or fused with animage generated with, for example, the structured light laparoscope togenerate a virtual shadow augmented image.

The introduction of virtual shadows into images generated by alaparoscope enables surgical procedures to be performed faster and moreefficiently with fewer errors compared to prior art techniques forimproving depth perception when viewing laparoscopic images and/orvideos. The introduction of virtual shadows also makes learninglaparoscopic surgical techniques and skills easier.

FIG. 1 shows a flowchart of one embodiment of a method 100 foraugmenting an image of a surface of an object with a virtual image. Themethod 100 may be performed using an imaging system shown in FIGS. 8 and9 or another imaging system. The method is implemented in the ordershown, but other orders may be used. Additional, different, or feweracts may be provided. Similar methods may be used for augmenting animage with a virtual shadow.

In act 102, a processor identifies an image of the surface of the objectand a tool being used during a medical procedure. Within the image, thetool is adjacent to the surface of the object. For example, the medicalprocedure may be a laparoscopic surgery. Laparoscopic surgery is aminimally invasive surgery that includes operations within abdominal orpelvic cavities. During the laparoscopic surgery, a laparoscope and/or atool are inserted into the object (e.g., a body of a patient). The toolmay be any number of medical devices including, for example, a trocar,forceps, scissors, a probe, a dissector, hooks, retractors, or othermedical devices.

In one embodiment, identifying the image of the surface of the objectincludes a first endoscope generating 2D image data representing thesurface of the object. The first endoscope may be any number ofdifferent types of endoscopes including, for example, a structured lightendoscope, a monocular laparoscope, a stereo scope, or another imagingdevice. The first endoscope may be inserted into the patient, forexample, via a small incision and a port installed at the incision. Alight source as part of the first endoscope or a separate light sourcemay also be introduced into the patient to illuminate the tissue to beimaged. In another embodiment, a plurality of images of the surface ofthe object are stored in a memory in communication with the processor,and the processor identifies the image of the surface of the object fromthe plurality of images of the surface stored in the memory.

FIG. 2 is an example of an image of a surface of an object 200 and atool 202 being used during a medical procedure. The object 200 may, forexample, be any number of organs within a human body. For example, theobject 200 may be the uterus, spleen, gallbladder, or another organwithin the abdomen of the patient. As shown in FIG. 2, the imagegenerated by the first endoscope, for example, is essentially a 2D imageof a 3D anatomy. With 2D images, the sense of depth and 3D is lost,which makes navigating difficult for a user (e.g., a surgeon). As shownin FIG. 2, the image generated by the first endoscope includes a faintshadow or no shadow at all.

In act 104, the processor identifies a 3D model of the surface of theobject. In one embodiment, identifying the 3D model of the surface ofthe object includes a second endoscope generating data representing thesurface of the object. The second endoscope may be any number of typesof endoscopes including, for example, a structured light laparoscope.Other types of endoscopes such as, for example, a stereo scope or amonocular laparoscope may be used to generate the data representing thesurface of the object. In one embodiment, the second endoscope and thefirst endoscope are the same endoscope. For example, the structuredlight endoscope may include a camera that operates at 60 frames/s. Theimage from act 102 is generated every other frame, and the data for the3D model from act 102 is generated the other frames.

In one embodiment, the structured light laparoscope, for example,generates a point cloud of the surface of the object. The point cloudrepresents 3D coordinates of points on an external surface of theobject. The structured light laparoscope measures a large number ofpoints on the external surface of the object and outputs the point cloudas a data file.

The processor generates a mesh of the surface of the object based on thepoint cloud generated by the structured light laparoscope. The processorgenerates the 3D model of the surface of the object based on thegenerated mesh. In one embodiment, the processor generates the 3D modelof the surface from a viewpoint from which the image of the surface ofthe object and a tool identified in act 102 is imaged. In other words,the 3D model of the surface is generated such that the 3D model of thesurface and the image identified in act 102 are from the same viewpoint.

In act 106, the processor generates a first augmented 3D model. Thegenerating of the first augmented 3D model includes superimposing arepresentation of the tool on the identified 3D model of the surface ofthe object. The processor identifies a position of the tool and therepresentation of the tool.

In one embodiment, an optical tracking system identifies a 3D positionof the tool relative to the 3D model of the surface of the object. Theoptical tracking system includes one or more cameras positioned at adistance away from the object being imaged. In one embodiment, theoptical tracking system includes two or more cameras that provideoverlapping projections. The processor uses image data generated by thetwo cameras to triangulate, for example, the 3D position of the toolrelative to the 3D model of the surface of the object.

The optical tracking system may include one or more (e.g., four) markerspositioned on the tool. In one embodiment, the optical tracking systemmay also include one or more (e.g., four) markers positioned on, forexample, the structured light endoscope. The four markers, for example,may be optical markers that are visible within images generated by thecameras. The one or more markers positioned on the tool and/or the oneor more markers positioned on the structured light endoscope may be anynumber of sizes and shapes. For example, the one or more markerspositioned on the tool and the one or more markers positioned on thestructured light endoscope are spherical and are large enough to bevisible within images generated by the two or more cameras. In otherembodiments, the 3D position of the tool is identified in other ways.For example, a tracking system identifies a 3D position of the toolrelative to the 3D model of the surface of the object withelectromagnetic (EM) trackers, mechanical trackers, shape sensors, orother types of trackers instead of or in addition to the opticaltracking system.

The processor triangulates the 3D position (e.g., 3D coordinates withina 3D space) of the tool and/or the 3D position (e.g., 3D coordinateswithin a 3D space) of the structured light endoscope based on image datagenerated by the two cameras, which includes representations of themarkers on the tool and/or the markers on the structured lightendoscope, and representations of markers with known locations withinthe field of view of the cameras. The 3D position of the tool and the 3Dposition of the structured light endoscope, for example, may be 3Dpositions of the markers on the tool and 3D positions of the markers onthe structured light endoscope, respectively. The processor may alsodetermine an orientation of the tool and/or an orientation of thestructured light endoscope based on relative orientations of the markersof the tool and the markers of the structured light endoscope,respectively. In one embodiment, the tool and/or the structured lightendoscope do not include markers, and the processor determines the 3Dposition of the tool and/or the 3D position of the structured lightendoscope based on surface features of the tool and/or the structuredlight endoscope, respectively. In other embodiments, other methods andsystems may be used to determine the 3D position of the tool and/or the3D position of the structured light endoscope. For example, the 3Dposition of the tool and the 3D position of the structured lightendoscope may be estimated directly from images or structured lightimages using tool detection.

The processor may identify the tool and/or the structured lightendoscope being used in, for example, the laparoscopic surgery. In oneembodiment, a user identifies a type of the tool and/or a type of thestructured light endoscope via, for example, one or more inputs (e.g., akeyboard and a mouse). In another example, the processor automaticallyidentifies the tool and/or the structured light endoscope being used inthe laparoscopic surgery, for example, based on data (e.g., identifyingdata) received from the tool and/or data (e.g., identifying data)received from structured light endoscope, respectively, or the imagedata generated by the cameras.

For example, the processor may match image data, which is generated bythe cameras and includes a representation of the tool and/or arepresentation of the structure light endoscope, with stored datacorresponding to different types of tools and/or endoscopes. Theprocessor may match characteristics of the tool and/or characteristicsof the structured light endoscope within the generated image data tocharacteristics of tools and/or characteristics of endoscopes within thestored data, respectively, to identify the type of the tool and/or thetype of the endoscope being used. For example, a particular endoscope(e.g., the structured light endoscope) may include a unique combinationof number of markers and size and markers. The processor may identify anumber of markers and a size of the markers on an endoscope (e.g.,characteristics) within image data generated by the cameras, and mayidentify that the particular structured light endoscope is being usedbased on a comparison of the identified characteristics withdescriptions of a plurality of different types of endoscopes (e.g.,different manufacturers, different models). For example, the memory maystore data describing the plurality of different types of endoscope, andthe stored data describing the particular structured light endoscope mayspecify that the particular structured light endoscope includes fourmarkers of a particular size.

Based on the identified type of the tool and the identified position andorientation of the tool, the processor may determine a 3D position of aproximal end of the tool in the 3D space. For example, with the knowntype of the tool, a length of the tool may be identified (e.g., from thestored data corresponding to different types of tools), and theprocessor may determine the 3D position of the proximal end of the toolin the 3D space based on the identified length of the tool and theidentified 3D positions of the markers on the tool. Further, based onthe identified type of the endoscope and the identified position andorientation of the endoscope, the processor may determine a 3D positionof a proximal end of the endoscope (e.g., the structured lightendoscope) within the 3D space.

The processor identifies a representation (e.g., a 3D model) of the toolbased on the identified type of the tool. In one embodiment, the memorystores a plurality of 3D models for respective tools, and the processoridentifies the 3D model of the tool from the plurality of stored 3Dmodels based on the identified type of the tool. For example, theprocessor identifies the tool being used within the laparoscopic surgeryis a particular trocar based on characteristics identified within theimage data generated by the cameras. The processor selects a 3D modelfor the particular trocar from the plurality of stored 3D models forrespective tools based on the identification of the tool. In otherembodiments, the processor generates the 3D model of the tool in realtime based on the determined 3D position of the tool, the determinedorientation of the tool, the identified type of the tool, stored datarelated to the identified tool (e.g., characteristics of the tool suchas size and shape of the tool), other information related to theidentified tool, or any combination thereof.

In another embodiment, the processor determines a shape and a size ofthe tool based on the image data generated by the cameras, an endoscope,or another imaging device (e.g., from act 102). The processor generatesthe 3D model of the tool based on the determined shape and thedetermined size of the tool.

The processor may also determine a position of the tool relative to theendoscope (e.g., the structured light endoscope). The processor maydetermine the position of the tool relative to the structured lightendoscope based on the 3D position of the tool and the 3D position ofthe structured light endoscope determined within the same 3D space. Forexample, the 3D position of the structured light endoscope may besubtracted from the 3D position of the tool. Based on the determinedposition of the tool relative to the structured light endoscope and theviewpoint from which the 3D model is generated in act 104, the processormay also determine the 3D position of the tool relative to the 3D modelof the surface of the object.

The processor generates the first augmented 3D model by fusing the 3Dmodel of the tool with the 3D model of the surface of the object basedon the determined 3D position of the tool relative to the 3D model ofthe surface of the object. In one embodiment, the 3D model of the tooland the 3D model of the surface of the object are already registered. Inanother embodiment, the 3D model of the tool and the 3D model of thesurface of the object are registered in any number of ways including,for example, with 3D/3D registration.

FIG. 3 shows an example of a first augmented 3D model 300. The firstaugmented 3D model 300 includes a 3D model 302 of the surface of theobject identified in act 104 and a 3D model 304 of the tool generated inact 106. The 3D model 302 of the surface of the object shown in FIG. 3includes a textured representation of the surface of the object. FIG. 4shows another example of a first augmented 3D model 400. The firstaugmented 3D model 400 includes a 3D model 402 of the surface of theobject identified in act 104 and a 3D model 404 of the tool generated inact 106. The 3D model 402 of the surface of the object shown in FIG. 4includes an untextured representation of the surface of the object.

In act 108, the processor introduces a virtual light source into thefirst augmented 3D model generated in act 106. The virtual light sourcemay be introduced into the first augmented 3D model in an up vector fromthe image identified in act 102. In one embodiment, the virtual lightsource is introduced into the first augmented 3D model at a positionupwards of gravity, as defined by the optical tracking system. Inanother embodiment, the virtual light source is introduced into thefirst augmented 3D model, such that the virtual light source isco-located with a real light source. The virtual light source thuscreates shadows in a similar configuration to real shadows that arefaint or not visible in the image identified in act 102, for example.This has the effect of enhancing the existing faint shadow.

In act 110, the processor generates a second augmented 3D model. Theprocessor generates a representation of a shadow on the surface of theobject in the generated first augmented 3D model based on the virtuallight source introduced into the generated first augmented 3D model. Thesecond augmented 3D model includes the fused 3D model of the tool andthe 3D model of the surface of the object and the generatedrepresentation of the shadow.

FIG. 5 shows an example of a virtual light source 500 introduced intothe example of the first augmented 3D model 400 shown in FIG. 4. Theprocessor generates the representation of the shadow 502 based on thevirtual light source 500 introduced into the first augmented 3D model inact 108. The processor estimates the shadow 502 that is cast on thesurface of the object within the first augmented 3D model 400 by the 3Dmodel 404 of the tool.

In act 112, the processor augments the identified image of the surfaceof the object. The processor subtracts the first augmented 3D modelgenerated in act 106 from the second augmented 3D model generated in act110. After the subtraction, a representation of a virtual shadowremains. FIG. 6 shows an example of a representation of a virtual shadow600 after the first augmented 3D model has been subtracted from thesecond augmented 3D model.

In one embodiment, a darkness and a transparency of the representationof the virtual shadow 600 is controllable via a user interface (e.g., aGUI) to identify a working region of the tool. As the tool gets closerto the surface of the object, the shadow formed by the tool gets darker.A darker shadow may occlude information on the surface of the object.Accordingly, the processor may lighten the representation of the shadow,or the representation of the shadow may be visualized with a border ofthe shadow.

The augmenting of the identified image of the surface of the objectincludes superimposing the generated representation of the shadow on theimage of the surface of the object and the tool identified in act 102.For example, the generated representation of the shadow may be fusedwith the image identified in act 102. In one embodiment, therepresentation of the virtual shadow is rendered separately and blended.

In one embodiment, the image of the surface of the object and the toolidentified in act 102 and the representation of the virtual shadow arealready registered since the 3D model of the surface of the objectidentified in act 104 and the image identified in act 102 are from thesame viewpoint. In another embodiment, the image of the surface of theobject and the tool identified in act 102 and the representation of thevirtual shadow are registered in any number of ways including, forexample, with 2D/2D registration.

FIG. 7 shows an example of a generated representation of a virtualshadow 700 fused with an image 702 of a surface of an object and a tool(e.g., the image identified in act 102). The introduction of the virtualshadow 700 into the image 702 of the surface of the object and the toolprovides augmented reality. With the use of, for example, a structuredlight laparoscope, a 3D model of the surface of the object and thus therepresentation of the shadow may be generated in real time. Theintroduction of the representation of the shadow into laparoscopicimages and/or video in real time enables surgical procedures to beperformed faster and more efficiently with fewer errors. Also, theintroduction of the representation of the shadow may make learninglaparoscopic surgical techniques and skills easier.

FIG. 8 shows one embodiment of an image-guided system 800. Theimage-guided system 800 may be used in the method described above andthe system described below. The image-guided system 800 may include oneor more imaging systems 802 (e.g., an endoscopy system), one or moreimage processing systems 804 (e.g., an image processing system), and oneor more tools 806 (e.g., a tool). A dataset representing atwo-dimensional (2D) or a three-dimensional (3D) (e.g., volumetric)region may be acquired using the imaging device 802 and the imageprocessing system 804 (e.g., an imaging system). The 2D dataset or the3D dataset may be obtained contemporaneously with the panning and/orexecution of a medical treatment procedure or at an earlier time.Additional, different or fewer components may be provided.

In one embodiment, the imaging system 802, 804 is, for example, anendoscope. The imaging system 802, 804 may be used to create a patientmodel that may be used during a medical procedure (e.g., a laparoscopicsurgery). For example, the image processing system 804 is a workstationfor laparoscopic surgery within the abdomen of a patient. In otherembodiments, the imaging system 102, 104 may include, for example, amedical workstation, a magnetic resonance imaging (MRI) system, acomputed tomography (CT) system, an ultrasound system, a positronemission tomography (PET) system, an angiography system, a fluoroscopy,an x-ray system, any other now known or later developed imaging system,or any combination thereof. The workstation 804 receives datarepresenting or images of the patient (e.g., including at least part ofthe abdomen of the patient) generated by the imaging device 802.

The tool 806 may be image guided by the imaging system 802, 804. Thetool 806 may be any number of tools including, for example, a trocar,forceps, scissors, a probe, a dissector, hooks, retractors, or othermedical devices. The tool 806 may be used, for example, to image, closewounds, take biopsies, cut, drain fluids, or grasp tissue. The tool 806may be image guided to facilitate diagnosis or treatment. The position,transmission, or other operation of the tool 806 may be controlled bythe image processing system 804 or another controller. The image-guidedsystem 800 may include more or fewer components.

FIG. 9 shows one embodiment of an imaging system 900. The imaging system900 includes the endoscopy system 802 and the image processing system804. The endoscopy system 802 includes one or more ports 902 (e.g., twoports), an endoscope 904, an optical tracking system 906, and a patientbed 908. The image processing system 804 may include a receiver 910, aprocessor 912, a memory 914, and a display 916. Additional, different,or fewer components may be provided. For example, an additionalendoscope may be provided for imaging. Additionally, a user input device918 (e.g., a keyboard and/or a mouse) may be provided for user control.

In one embodiment, the processor 912 and the memory 914 are part of theendoscopy system 802. Alternatively, the processor 912 and the memory914 are part of an archival and/or image processing system, such asassociated with a medical records database workstation or server. In yetother embodiments, the processor 912 and the memory 914 are a personalcomputer such as a desktop or a laptop, a workstation, a server, anetwork, or combinations thereof. The processor 912 and the memory 914may be provided without other components for implementing the method.

The patient bed 908 (e.g., a patient gurney or table) supports anexamination subject 920 such as, for example, a patient. In order toexamine and/or operate on the patient, the endoscope 904 is insertedinto one of the ports 902, and the endoscope 904 generates image datafor a portion of the patient within a field of view. The endoscope 904may include any number of parts including, for example, a tube (e.g., arigid or flexible tube), a light delivery system to illuminate theportion of the patient, a lens system, an eyepiece, a camera, or anycombination thereof. In one embodiment, the endoscope 904 is astructured light endoscope that illuminates the portion of the patientvia fiberoptics, and generates image data with a camera. The endoscope904 generates and transmits image data to the receiver 910.

The tool 806 is inserted into another one of the ports 902. In oneembodiment, a hollow cannula 922 is inserted into the other port 902,and the tool 806 is inserted into the hollow cannula 922. The tool 806is guided within the patient based on images generated by the endoscope904 or another endoscope.

The endoscope 904 connects with the receiver 910. The connection iswired (e.g., using a coaxial cable) or wireless. The connection is forimage data from the endoscope 904 to be transmitted to and received bythe receiver 910. The receiver 910 includes the processor 912 or anotherprocessor (e.g., a digital signal processor, a field programmable gatearray, or an application specific circuit for applying an inverseFourier transform) for generating an image based on the image datagenerated by the endoscope 904. The receiver 910 is configured byhardware or software.

The processor 912 is a general processor, a central processing unit, acontrol processor, a graphics processor, a digital signal processor, athree-dimensional rendering processor, an image processor, anapplication-specific integrated circuit, a field-programmable gatearray, a digital circuit, an analog circuit, combinations thereof, orother now known or later developed device for image processing. Theprocessor 912 is a single device or multiple devices operating inserial, parallel, or separately. The processor 912 may be a mainprocessor of a computer, such as a laptop or desktop computer, or may bea processor for handling some tasks in a larger system, such as beingpart of the receiver 910 or the imaging system 804. The processor 912 isconfigured by instructions, design, hardware, and/or software to performthe acts discussed herein, such as augmenting an image of a surface.

The memory 914 is a computer readable storage media. The computerreadable storage media may include various types of volatile andnon-volatile storage media, including but not limited to random accessmemory, read-only memory, programmable read-only memory, electricallyprogrammable read-only memory, electrically erasable read-only memory,flash memory, magnetic tape or disk, optical media and the like. Thememory 914 may be a single device or a combination of devices. Thememory 914 may be adjacent to, part of, networked with and/or remotefrom the processor 912.

The display 916 is a monitor, a CRT, an LCD, a plasma screen, a flatpanel, a projector or other now known or later developed display device.The display 916 is operable to generate images for a two-dimensionalview or a rendered three-dimensional representation. For example, atwo-dimensional image representing a three-dimensional volume throughrendering is displayed.

The optical tracking system 906 includes one or more (e.g., two) cameras924, first markers 926 (e.g., optical markers), and/or second markers928 (e.g., optical markers). The two cameras 924, for example, arepositioned, such that the first markers 926 and the second markers 928are within the field of view of the two cameras 924. The cameras 924 maybe fixed or positionable such that the first markers 926 and the secondmarkers 928 are within the field of view of the two cameras 924.

The first markers 926 are positioned on the endoscope 904, and thesecond markers 928 are positioned on the tool 806. The first markers 926and the second markers 928 may be any number of sizes and/or shapes. Forexample, the first markers 926 and the second markers 928 may bespherical in shape and may be sized such that the first markers 926 andthe second markers 928 are visible within images generated by thecameras 924. Other shapes and/or sizes may be provided. In theembodiment shown in FIG. 9, the first markers 926 include four firstmarkers 926, and the second markers 928 include four second markers 928.More or fewer first markers 926 and/or second markers 928 may beprovided. The first markers 926 and the second markers 928 may bepositioned in any number of patterns relative to each other (e.g.,x-shaped pattern) and/or may be positioned in any number of locationsalong the endoscope 904 and the tool 806, respectively.

In one embodiment, the memory 914 is configured to store an image of asurface (e.g., an internal surface) of, for example, the patient 920 andat least a portion of the tool 806. The memory is also configured tostore a 3D model of the surface of the patient 920. The processor 912 isconfigured to superimpose a representation of the tool 806 on theidentified 3D model of the surface of the patient 920. The processor 912is also configured to introduce a virtual light source into thesuperimposed representation of the tool 806 and identified 3D model ofthe surface of the patient 920, for example. The processor 912 isconfigured to generate a representation of a shadow on the surface ofthe patient 920 within the superimposed representation of the tool 806and identified 3D model of the patient 920 based on the introduced lightsource. The processor 912 is configured to superimpose the generatedrepresentation of the shadow on the stored image of the surface of thepatient 920 and at least the portion of the tool 806.

In one embodiment, one or more endoscopes (e.g., the endoscope 904) areconfigured to generate first data representing the surface of theobject, and generate second data representing the surface of the object.The processor 912 is further configured to generate the 3D model of thesurface of the patient 920 based on the generated first data, andgenerate the image of the surface of the patient 920 and at least theportion of the tool 806 based on the generated second data.

In one embodiment, the optical tracking system 906 is configured todetermine a 3D position of the tool 806 relative to the 3D model of thesurface of the patient 920. The one or more cameras 924 are positionedat a distance away from the patient 920. The one or more cameras 924 areconfigured to generate image data for the first markers 926 positionedon the endoscope 904 and the second markers 928 positioned on the tool806. The processor 912 is further configured to identify a 3D model ofthe tool 806 from a plurality of predetermined models of tools stored inthe memory 914 based on the tool 806 (e.g., a type of tool) being used.Alternatively, the processor 912 is configured to determine a shape anda size of the tool 806 based on the image of the surface of the patient920 and at least the portion of the tool 806 stored in the memory 914.The processor 912 is further configured to generate the 3D model of thetool 806 based on the determined shape and the determined size of thetool 806.

The processor 912 is configured to identify the 3D position of the tool806 relative to the 3D model of the surface of the patient 920. Theidentification of the 3D position of the tool 806 relative to the 3Dmodel of the surface of the patient 920 includes estimation of the 3Dposition of the tool 806 relative to the 3D model of the surface of thepatient 920 based on image data for the first markers 926 positioned onthe endoscope 904 and the second markers 928 positioned on the tool 806,and the identified model of the tool 806. The processor 912 is furtherconfigured to superimpose the representation of the tool 806 on theidentified 3D model of the surface of the patient 920 based on theidentified 3D position of the tool relative to the 3D model of thesurface of the object.

While the present invention has been described above by reference tovarious embodiments, it should be understood that many changes andmodifications can be made to the described embodiments. It is thereforeintended that the foregoing description be regarded as illustrativerather than limiting, and that it be understood that all equivalentsand/or combinations of embodiments are intended to be included in thisdescription.

1. A method for augmenting an image of a surface of an object, themethod comprising: identifying, by a processor, the image of the surfaceof the object, the image of the surface of the object being an image ofthe surface of the object and at least a portion of a tool; identifying,by the processor, a three-dimensional (3D) model of the surface of theobject; generating, by the processor, a first augmented 3D model, thegenerating of the first augmented 3D model comprising superimposing arepresentation of the tool on the identified 3D model of the surface ofthe object; introducing, by the processor, a virtual light source intothe generated first augmented 3D model; generating, by the processor, asecond augmented 3D model, the generating of the second augmented 3Dmodel comprising generating a representation of a shadow on the surfaceof the object in the generated first augmented 3D model based on thevirtual light source introduced into the generated first augmented 3Dmodel; and augmenting, by the processor, the identified image of thesurface of the object, the augmenting of the identified image of thesurface of the object comprising superimposing the generatedrepresentation of the shadow on the identified image of the surface ofthe object and the tool.
 2. The method of claim 1, wherein identifyingthe image of the surface of the object comprises: generating, by alaparoscope, image data representing the surface of the object; andgenerating, by the processor, the image of the surface of the objectbased on the generated image data.
 3. The method of claim 1, whereinidentifying the 3D model of the surface of the object comprises:generating, by an endoscope, data representing the surface of theobject; and generating, by the processor, the 3D model of the surface ofthe object based on the generated data.
 4. The method of claim 3,wherein the identified image of the surface of the object is from aviewpoint, and wherein generating the 3D model of the surface of theobject comprises generating the 3D model of the surface of the objectbased on the generated data from the same view point.
 5. The method ofclaim 3, wherein the endoscope is a structured light endoscope, whereingenerating the data representing the surface of the object comprises:generating, with the structured light endoscope, a point cloud of thesurface of the object; and generating, with the processor, a mesh of thesurface of the object based on the generated point cloud, and whereingenerating the 3D model of the surface of the object based on thegenerated data comprises generating the 3D model of the surface of theobject based on the generated mesh of the surface of the object.
 6. Themethod of claim 5, further comprising identifying the position of thetool.
 7. The method of claim 6, wherein identifying the position of thetool comprises identifying, by an optical tracking system, a 3D positionof the tool relative to the 3D model of the surface of the object. 8.The method of claim 7, wherein the optical tracking system comprises:one or more markers positioned on the tool; one or more markerspositioned on the structured light endoscope; and one or more cameraspositioned at a distance away from the object, the one or more camerasbeing configured to generate image data for the one or more markers ofthe tool and the one or more markers of the structured light endoscope,wherein the method further comprises identifying, by the processor, a 3Dmodel of the tool, and wherein identifying the 3D position of the toolrelative to the 3D model of the surface of the object comprisesestimating the 3D position of the tool relative to the 3D model of thesurface of the object based on the image data for the one or moremarkers of the tool and the one or more markers of the structured lightendoscope, and the identified model of the tool.
 9. The method of claim7, wherein identifying the 3D model of the tool comprises: identifying,by the processor, the 3D model of the tool from a plurality ofpredetermined models of tools stored in a memory based on the tool beingused, the memory being in communication with the processor; ordetermining, by the processor, a shape and a size of the tool based onthe identified image of the surface of the object and the tool, andgenerating, by the processor, the 3D model of the tool based on thedetermined shape and the determined size of the tool.
 10. The method ofclaim 1, wherein introducing the virtual light source into the generatedfirst augmented 3D model comprises introducing the virtual light sourceinto the generated first augmented 3D model in an up vector from theidentified image, or at a position upwards of gravity, as defined by anoptical tracking system configured to identify a position of the tool.11. The method of claim 1, further comprising subtracting, by theprocessor, the generated first augmented 3D model from the generatedsecond augmented 3D model, such that the representation of the shadowremains.
 12. The method of claim 11, wherein augmenting the identifiedimage of the surface of the object comprises blending or fusing therepresentation of the shadow with the identified image of the surface ofthe object.
 13. In a non-transitory computer-readable storage mediumstoring instructions executable by one or more processors to augment animage of a surface of an object, the instructions comprising:identifying, by a processor, the image of the surface of the object, theimage of the surface of the object being an image of the surface of theobject and at least a portion of a tool; identifying, by the processor,a three-dimensional (3D) model of the surface of the object;superimposing, by the processor, a representation of the tool on theidentified 3D model of the surface of the object; introducing, by theprocessor, a virtual light source into the superimposed representationof the tool and identified 3D model of the surface of the object;generating, by the processor, a representation of a shadow on thesurface of the object within the superimposed representation of the tooland identified 3D model of the surface of the object based on theintroduced light source; and superimposing, by the processor, thegenerated representation of the shadow on the identified image of thesurface of the object and the tool.
 14. The non-transitorycomputer-readable storage medium of claim 13, wherein the surface of theobject is an internal surface of a body of a patient.
 15. Thenon-transitory computer-readable storage medium of claim 13, wherein theinstructions further comprise identifying the position of the tool, theidentifying of the position of the tool comprising identifying, by anoptical tracking system, a 3D position of the tool relative to the 3Dmodel of the surface of the object.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein the opticaltracking system comprises: one or more markers positioned on the tool;one or more markers positioned on the structured light endoscope; andone or more cameras positioned at a distance away from the object, theone or more cameras being configured to generate image data for the oneor more markers of the tool and the one or more markers of thestructured light endoscope, wherein the instructions further compriseidentifying, by the processor, a 3D model of the tool, and whereinidentifying the 3D position of the tool relative to the 3D model of thesurface of the object comprises estimating the 3D position of the toolrelative to the 3D model of the surface of the object based on the imagedata for the one or more markers of the tool and the one or more markersof the structured light endoscope, and the identified model of the tool.17. The non-transitory computer-readable storage medium of claim 15,wherein identifying the 3D model of the tool comprises: identifying, bythe processor, the 3D model of the tool from a plurality ofpredetermined models of tools stored in a memory based on the tool beingused, the memory being in communication with the processor; ordetermining, by the processor, a shape and a size of the tool based onthe identified image of the surface of the object and the tool, andgenerating, by the processor, the 3D model of the tool based on thedetermined shape and the determined size of the tool.
 18. A system forgenerating an augmented image of a surface of an object and at least aportion of a tool, the system comprising: a memory configured to store:an image of the surface of the object and at least the portion of thetool; and a three-dimensional (3D) model of the surface of the object;and a processor in communication with the memory, the processor beingconfigured to: superimpose a representation of the tool on theidentified 3D model of the surface of the object; introduce a virtuallight source into the superimposed representation of the tool andidentified 3D model of the surface of the object; generate arepresentation of a shadow on the surface of the object within thesuperimposed representation of the tool and identified 3D model of thesurface of the object based on the introduced light source; andsuperimpose the generated representation of the shadow on the storedimage of the surface of the object and at least the portion of the tool.19. The system of claim 18, further comprising: one or more endoscopesconfigured to: generate first data representing the surface of theobject; and generate second data representing the surface of the object,wherein the processor is further configured to: generate the 3D model ofthe surface of the object based on the generated first data; andgenerate the image of the surface of the object and at least the portionof the tool based on the generated second data.
 20. The system of claim19, further comprising an optical tracking system configured todetermine a 3D position of the tool relative to the 3D model of thesurface of the object, the optical tracking system comprising: one ormore markers positioned on the tool; one or more markers positioned onthe endoscope; and one or more cameras positioned at a distance awayfrom the object, the one or more cameras being configured to generateimage data for the one or more markers of the tool and the one or moremarkers of the endoscope, wherein the processor is further configuredto: identify a 3D model of the tool, the identification of the 3D modelof the tool comprising: identification of the 3D model of the tool froma plurality of predetermined models of tools stored in the memory basedon the tool being used; or determination of a shape and a size of thetool based on the stored image of the surface of the object and at leastthe portion of the tool, and generation of the 3D model of the toolbased on the determined shape and the determined size of the tool; andidentify the 3D position of the tool relative to the 3D model of thesurface of the object, the identification of the 3D position of the toolrelative to the 3D model of the surface of the object comprisingestimation of the 3D position of the tool relative to the 3D model ofthe surface of the object based on the image data for the one or moremarkers of the tool and the one or more markers of the structured lightendoscope, and the identified model of the tool, and wherein theprocessor is configured to superimpose the representation of the tool onthe identified 3D model of the surface of the object based on theidentified 3D position of the tool relative to the 3D model of thesurface of the object.